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MASSACHUSETTS 


ABSTRACT 


An  ^almost  noiseless*  channel  is  defined.  The  definition  corresponds  to 

many  physical  channels  operating  at  high  signal-to-noise  ratio.  Attention  is 

focused  on  the  decrement  in  capacity  and  computation  cutoff  rate  as  the  noise 

increases  from  zero.  It  is  shown  that  the  decrement  in  R-  is  dominated  by 

’comp 

a  term  proportional  to  the  square  root  of  the  noisiest  transition  probability. 


/ 


■a,  to  a  nominally  noiseless  output.  In  contrast,  the  decrement  in  C  in  many 
cases  of  interest  is  dominated  by  a  term  proportional  tov-alno.  Accordingly 

J  ' — *1  /*— 

the  decrement  in  can  si8ni-f  icantly  exceed  the  decrement  in  C. 

This  paper  was  presented  at  the  1979  IEEE  International  Symposium  on 
Information  Theory  held  in  Grignano,  Italy  on  25-29  June  1979. 
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I.  INTRODUCTION 

This  paper  addresses  the  subject  of  memoryless  channels  which  are  "almost 
noiseless,"  i.e.,  channels  where  errors,  suitably  defined,  are  infrequent. 

More  specifically,  the  capacity  and  Rcomp  behavior  of  these  channels  are 
addressed  in  terms  of  the  noise  characteristics. 

This  obscure  theme  is  motivated  by  some  practical  problems  in  the 
engineering  design  of  satellite  communications  systems.  The  context  is  given 
in  Fig.  1.  The  satellite  is  not  the  familiar  frequency  translating  repeater. 
Rather  it  is  a  "signal  processing  satellite"  in  which  one  or  more  uplinks  are 
demodulated  into  bits  which  are  then  modulated  onto  a  downlink  carrier  and 
beamed  towards  the  receive  terminal. 

The  motivations  for  performing  this  on-board  signal  processing  are  several: 
i)  to  avoid  transmitting  uplink  noise  or  interference  on  the  downlink  as  would 
be  the  case  with  a  frequency  translating  repeater,  and  ii)  to  permit  the 
decoupling  of  uplink  and  downlink  modulation  formats  to  better  match  hardware 
constraints.  For  example,  it  may  be  desirable  to  operate  several  individual 
uplinks  FDMA  and  TDM  on  a  single  downlink  carrier. 

This  situation  is  modelled  as  shown  in  Fig.  2.  Note  that  the  output 
of  the  uplink  channel  is  directly  connected  to  the  input  of  the  downlink 
channel  without  benefit  of  any  decoding/reencoding  operation.  This  constraint 
is  associated  with  the  desire  to  avoid  the  satellite  complexity  implied  by  any 
substantive  on-board  decoding.  System  error  control  is  accomplished  via  end- 
to-end  coding,  i.e.,  encoding  takes  place  at  the  transmit  terminal  and  decoding 
takes  place  at  the  receive  terminal. 

The  engineering  design  of  such  a  system  entails  a  number  of  complex  trade¬ 
offs.  A  useful  initial  design  approach  is  to  specify  the  uplink  modulation 
format  and  the  end-to-end  code  assuming  the  downlink  is  noiseless.  One  would 
then  perturb  this  first  cut  design  to  the  extent  that  the  downlink  departs 
from  the  noiseless  condition. 

Hopefully  this  motivates  an  understanding  of  almost  noiseless  channels. 
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SATELLITE 
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Fig.  1.  Communication  with  a  processing  satellite. 
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Fig.  2.  Processing  satellite  model. 
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II.  DEFINITIONS 


A  discrete  memoryless  channel  is  defined  by 

(a)  an  input  alphabet,  X  =  {x^},  i=l,...,A 

(b)  an  output  alphabet,  Y  =  {y^},  j“l,..«,B 

(c)  a  transition  probability  matrix  {q..},  where  Z  q  =  1 

The  channel  is  defined  as  noiseless  if  and  only  if  q^  =  0  or  1  for  all 
i,j.  The  input  i  and  output  j  are  called  connected  if  q^  =  1;  i  and  j  are 
not  connected  if  q^  =  0. 

Suppose  b  £  B  of  the  outputs  are  connected  to  some  x^.  If  any  specific 
y^  is  connected  to  more  than  one  x^,  the  information  transmission  capability 
of  the  channel  is  not  impaired  if  these  several  x^  connected  to  the  same  y 
are  treated  as  a  single  input.  Thus,  for  noiseless  channels,  it  is  sufficient 
to  let  A  =  b  £  B. 

The  noiseless  channel  described  above  has  capacity,  C,  and  computa¬ 
tion  cutoff  rate,  R  ,  equal  to 
comp 

C  =  R  =  logA 
comp 

Capacity  is  achieved  when  the  A  inputs  are  equally  likely. 

The  noiseless  channel  described  above  is  the  limiting  form  of  an  ideal¬ 
ization  of  a  physical  channel  with  "very  large"  signal-to-noise  ratio. 
Actually,  for  finite  signal-to-noise  ratio,  the  channel  will  not  be  "noise¬ 
less".  This  paper  addresses  the  question  of  how  the  characteristics  of  the 
channel  change  when  "small"  noise  effects  are  present. 

An  almost  noiseless  channel  is  defined  as  one  with  a  transition 
probability  matrix  {q^}  in  the  form 


qij  =  +  aij  ’  (i"1*,,,,A5  J-1*--**85  A  -  B) 
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where 


f 4> ^ j }  is  the  transition  probability  matrix  of  a  noiseless  channel. 


{a„}  is  an  AxB  matrix  with  a^.«l  for  all  i»j« 

Since  q_  and  are  each  transition  probability  matrices,  it  is  easy  to 


show  that 


£a.  =  0  ,  i  1,  •  •  • ,  A 

j  J 


Since  {q^}  is  a  probability  matrix,  0  <_  q„  <_  1  for  all  {i,j}.  For 
x^  and  y^  not  connected  via  <J> ^ 0;  for  x^  and  y^  connected  via  <J> , 

“ij  i  °- 

It  will  be  convenient  in  the  sequel  to  define  several  sets: 


I  =  <vyjl*ij  =  1} 

II  =  (x1,yj|<t'kj  =  1  for  some  k  j  1} 

III  =  {x1,yJ|4'kj  =  0  for  k  =  1 . A} 

YNN  =  ^yj^ij  *  ^  ^°r  some  ^  (Nominally  Noiseless  outputs) 

Y^  ■  { y  J  |  $  i  j  -  0  for  i  -  1.....A}  (Complement  of  Y^) 


Clearly, 


I  0  II  U  III  ■  XY 


Y  U  Y  -  Y 
NN  NN 


Figure  3  illustrates  these  sets  with  an  example. 
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Fig.  3.  Sets. 
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I 

III.  CAPACITY  CONSIDERATIONS 

It  was  observed  in  Section  II  that  the  capacity  of  the  noiseless  channel 
is  obtained  with  equally  likely  inputs.  The  capacity  of  the  almost  noiseless 
channel  will  be  obtained  with  inputs  which  are  almost  equally  likely,  i.e., 
let 


Pi  =  A(1  +  ai} 


i  1 , .  • . ,  A 


(2) 


where 


oi<<l.  Clearly, 


la  =  0. 


(3) 


{pj  induces  a  distribution  on  Y  given  by 


r.  =  Ep^q,.  «  7  E(l+o.) 


‘j  "riMj  A  , 

j  i  j  x 


-U+o.  +  la±.  +  la^]  ,  Yj  CYnn 


A[^aij  +  ^“ij1  ’  yj  C  YNN 


(4) 


In  Eq.  (4),  the  convention  has  been  adopted  that  =  1  if  i= j ,  i.e.,  x^ 
is  connected  to  y  if  and  only  if  i=j . 

The  summation  in  Eq.  (4)  involves  the  product  of  small  numbers. 

Neglecting  this  second  order  term  results  in  the  following  approximation 


A[!  +  Oj  +  .  Yj  C  YM 


A  \a±i  ’  Yj  C  YNN 


(5) 
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Now 


I (X;Y)  =  H(Y)  -  H(Y|X) 


-  H(T)  -  rrjlogr.  ■  E  ±(1^  +  Ea1J)loi[i(14<,j  +  1.^)] 


NN  1  1 


Combining  terms  and  utilizing  Eqs.  (1)  and  (3), 


H(Y)  =  logA  -  j  Z  q  -  j  Z  (Zq  )log(Zq  ) 
A  I+II  13  A  i  ij  i  ±3 


In  going  from  Eq.  (6)  to  (7),  second  order  terms  were  neglected  as  before 
Note  that  H(Y)  is  independent  of  first-order  perturbations  in  the  input 
probabilities . 


-H(Y|X)  =  Z  p  q  logq 


-AH(Y|  X)  =  Z  (1-hj  )(*lj4t»  )log(*1  -h*  ) 

1 » j 

=  Z(l+ffi)(l-talj)log(l+alj) 

+  Z  (l+o  )a  loga 
II+III  3  3 


n  Sa-M  +  T  1  a^logCl 
*  I  lj  A  II+III  ij  ij 


where  the  final  approximation  has  neglected  second  order  terms.  Note  that 
H(Y|X)  is  also  independent  of  first  order  perturbations  in  the  input 
probabilities.  Combining  Eqs.  (7)  and  (8), 
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II 


ynn  1 


E  a 
II+III 


ijl08°ij 


(9) 


where,  in  Eq.  (9),  H(Y)-H(Y|x)  has  been  set  equal  to  capacity  since,  to 
first  order,  the  difference  does  not  vary  with  small  changes  in  the  input 
probabilities. 


+  Z  (la  )log(Eot  )  -  Z  a.  logoi.  ] 
Ym  i  3  i  J  II+III  ij 


(10) 


Note  that  AC  has  two  components,  one  from  set  II  and  one  from  set  III.  These 
components  can  be  written  as 


AC 


II 


ot  a 

f  y-i1’  log  '-i1' 


(11) 


ACm  "  A^-^j) 

ynn  1  i 


JJL 


(Ea 


ij 


T  log“u 1 


(12) 


Since  the  log  function  is  monotonically  increasing,  the  bracketed  expression 
in  Eq.  (12)  is  non-negative  for  all  j.  Accordingly,  AC^  and  AC^^.  are 
individually  non-negative.  Thus,  it  is  seen  that  the  capacity  decrement  is 
attributable  to  transitions  to  nominally  noiseless  outputs,  and  transitions  to 
not  nominally  noiseless  outputs,  with  the  two  kinds  of  transitions  contributing 
somewhat  differently. 

For  the  BSC,  the  exact  value  of  AC  is  -plogp  -  (l-p)log(l-p)  for 
0  £  p  <_  1.  The  approximate  value  of  AC  obtained  from  Eq.  (11)  is  -plogp 
+  p.  The  ratio  of  these  quantities  is  within  1%  of  unity  for  p  _<  0.07 
and  within  5%  of  unity  for  p  _<  0.22. 

_ 

Note  that  set  III  is  empty  for  the  BSC. 
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I 

For  the  BEC,  the  exact  value  of  AC  is  plog2  for  0  <_  p  <_  1.  The  approxi- 

* 

mate  value  obtained  from  Eq.  (12)  is  also  plog2.  Thus,  the  approximation 
is  exact  for  the  BEC. 

_ 

Note  that  set  II  is  empty  for  the  BEC. 
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CONSIDERATIONS 


IV.  R 


e  '“P  -  I  [I  P^q1:|l  -  I  WSj  \j 

J  1  1  >  J  »  K 

The  summation  on  j  will  contain  terms  of  the  form 


pipk/qij/qkj  =7(1‘kTi)  (1'K7k)(^ij4oij)1/2  (^kj'h3,kj)1/2 


— 2"  U+o.)  (1+aij)  ,  i  =  j  -  k 


^2  (1**,)  (l-HJk)  (1+aij)1/2c‘kj1/2.  ls^k 
^2  (1-HJi)  (l-HJk)aij1/2(l-Hikj)1/2,  i  *  j  -  k 

^2  d+O,)  (1-Vai:)1/2  “jk1/2*  1  *  j  *  k 


Expanding,  preserving  only  lowest  order  terms,  and  summing 


\2e  COmp  =  A  +  21  /a , 


Note  that  to  lowest  order,  R  is  also  independent  of  small  variations  in 

comp  r 

the  input  probabilities.  From  (13) 


l  =  log  - — - 

comp  1+2£/ 

A  n  ij 


log A  -  log [ 1  +  t  I  /a,.] 
A  XI  ij 


%  logA  -  j  z/ati 
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AR 

comp 


i  ]/•» 


(15) 


Note  that  the  decrement  in  R  is  dominated  by  transitions  to  nominally 

comp 

noiseless  outputs,  i.e.,  the  decrement  will  contain  terms  of  order  /a^ 

for  y,  C  Y„v,.  However,  for  y.  C  Y.„.,  the  decrement  contains  no  terms  of 
j  NN  j  NN 

order  /a^  . 

For  the  BSC,  the  exact  value  of  AR  is  log(l  +  2/p(l-p)  ).  The 

comp 

approximate  value  obtained  from  Eq.  (15)  is  2/p.  The  ratio  of  these  values 

-4  -3 

is  within  1%  of  unity  for  p  10  and  within  5%  of  unity  for  p  <  3  x  10 

2 

For  the  BEC,  the  exact  value  of  AR  =  log(l+2p)  =  2p  -  2p  +  ...  . 

comp 

Note  that  the  leading  term  is  proportional  to  p.  This  is  consistent  with 

/  * 

the  approximation  which  predicts  no  contribution  proportional  to  /p. 

* 

Recall  that  for  the  BEC,  II  is  empty. 
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V.  COMPARISON  OF  AC  AND  AR 

comp 

It  has  been  noted  that  AC  has  two  components:  one  contributed  by  Type  II 
transitions  and  given  by  Eq.  (11),  and  one  contributed  by  Type  III  transitions 
and  given  by  Eq.  (12).  On  the  other  hand,  ARcomp  as  expressed  in  Eq.  (15) 
contains  only  Type  II  transitions.  Forming  the  ratio  of  Eqs.  (11)  and  (15), 


AC 


II 


AR 


comp 


(  2 '  I  7fT 


ii 


ij 


where 


(16) 


(17) 


Corresponding  to  every  term  in  the  numerator  of  Eq.  (16)  is  a  term  in  the 
denominator.  The  ratio  of  these  terms  is 


logB., 

-i— J '  ‘  2'V°«  'hi 


(18) 


The  function  -wlogw  is  mouotonic  in  w  for  0  w  <_  — .  Thus  over  the  range 


of  interest. 


where 


-8,,logB 

—  la  <  -2/B  log/6 
/0  ~  max  max 

'hi 


6  =  max  (B  . .}  =  —  max  {a. . } 

max  i:t  ij  e  I;[  ij 


(19) 


(20) 


Combining  (16),  (17),  and  (18) 


AC 


II 


AR 


a  1/2  a  1/2 
/  .  max.  ,  ,  max. 

/e  (— — )  log(— — ) 


(21) 


comp 


Since  w  *  /B  “  /^,  w  <  —  corresponds  to  a  <  — . 
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The  right  hand  side  of  (21)  increases  from  zero  as  a  increases,  and  will 

-1/2  m3X 

always  be  less  than  its  maximum  value  of  e  for  a  <  1/e.  In  other  words, 

max  * 

for  an  almost  noiseless  channel,  the  decrement  in  R  always  exceeds  AC,.. 

comp  J  II 

For  a  channel  where  Y  =  Y,„,,  AC  =  ACTT  and  AR  >  AC.  For  a  channel 

NN  II  comp 

with  Ynn  not  empty,  ARcomp  nay  be  smaller  or  larger  than  AC  depending 
on  the  relative  magnitudes  of  {a„  II}  and  {a^  III}. 

Figure  4  displays  the  exact  and  approximate  expressions  for 
for  the  BSC.  comp 
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VI.  CONCLUSIONS 


For  "almost  noiseless"  discrete  memoryless  channels 

a)  C  is  achieved  and  R  is  maximized  with  equally  likely 
channel  inputs 

b)  the  decrement  in  C  arises  in  part  from  terms  of  the  form 
(-aloga),  where  a  <<  1  is  a  noisy  transition 


c)  the  decrement  in  R  arises  from  terms  of  the  form  /a 

comp 

d)  conclusions  b)  and  c)  imply  that  the  decrement  R  can 

comp 

significantly  exceed  the  decrement  in  C. 
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